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Abstract 

Background: Clinical vignettes have been used widely to compare quality of clinical care and to assess 
variation in practice, but the effect of different response formats has not been extensively evaluated. Our 
objective was to compare three clinical vignette-based survey response formats - open-ended 
questionnaire (A), closed-ended (multiple-choice) questionnaire with deceptive response items mixed 
with correct items (B), and closed-ended questionnaire with only correct items (C) - in rheumatologists' 
pre-treatment assessment for tumor-necrosis-factor (TNF) blocker therapy. 

Methods: Study design: Prospective randomized study. Setting: Rheumatologists attending the 2004 
French Society of Rheumatology meeting. Physicians were given a vignette describing the history of a 
fictitious woman with active rheumatoid arthritis, who was a candidate for therapy with TNF blocking 
agents, and then were randomized to receive questionnaire A, B, or C, each containing the same four 
questions but with different response formats, that asked about their pretreatment assessment. 
Measurements: Long (recommended items) and short (mandatory items) checklists were developed for 
pretreatment assessment for TNF-blocker therapy, and scores were expressed on the basis of responses 
to questionnaires A, B, and C as the percentage of respondents correctly choosing explicit items on these 
checklists. Statistical analysis: Comparison of the selected items using pairwise Chi-square tests with 
Bonferonni correction for variables with statistically significant differences. 

Results: Data for all surveys distributed (114 As, I 18 Bs, and I 18 Cs) were complete and available for 
analysis. The percentage of questionnaire A, B, and C respondents for whom data was correctly complete 
for the short checklist was 50.4%, 84.0% and 95.0%, respectively, and was 0%, 5.0% and 5.9%, respectively, 
for the long version. As an example, 65.8%, 85.7% and 95.8% of the respondents of A, B, and C 
questionnaires, respectively, correctly identified the need for tuberculin skin test (p < 0.0001). 

Conclusion: In evaluating clinical practice with use of a clinical vignette, a multiple-choice format rather 
than an open-ended format overestimates physician performance. The insertion of deceptive response 
items mixed with correct items in closed-ended (multiple-choice) questionnaire failed to avoid this 
overestimation. 
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Background 

Improvement of quality of clinical practice needs quality 
measurements. These measurements must be accurate, 
valid and feasible. The advantages and disadvantages of 
different methods of measuring the process of care, 
including both the competence of the clinician and what 
the clinician actually does, have been well described. 
Methods include chart extraction, standardized patients 
and clinical vignettes. Substantial inaccuracies in admin- 
istrative data are common, which leads to expensive data 
extraction and difficulty in validating data [1-3]. Com- 
pared to standardized patients and chart extraction, clini- 
cal vignettes are an accurate, valid, feasible and 
inexpensive tool to measure quality of health care [4,5]. 
Thus, clinical vignettes have been used widely to compare 
quality of clinical care and to assess variation in practice 
across countries, health care systems, specialties or clini- 
cians [6-11]. 

Vignette-based surveys for physicians feature open-ended 
questions rather than multiple-choice questions or check- 
lists. In this way, physicians can give a personal response 
to each question, which ensures that the survey captures 
the full range of practice variation [11,12]. Open-ended 
questions avoid the "cueing" inherent in multiple-choice 
questions, which could overestimate real physician per- 
formance. However, the accuracy of close-ended ques- 
tionnaires has been demonstrated in several domains: the 
format provides a significantly higher rate of accuracy 
than an open-ended format in terms of eyewitness confi- 
dence with the former format [ 1 3 ] . Close-ended question- 
naires maximize questionnaire response rate and ensure 
questionnaire completeness [14]. 

Moreover, different formats yield different answers: in one 
study, a closed-ended questionnaire produced results 
reflecting higher willingness to pay for a health care inter- 
vention, with different justifications for those evaluations 
than did those from an open-ended one [15]. A public 
opinion survey with two different response modalities 
asked subjects about the most important problem facing 
the United States: respondents of the open-ended format 
most often complained about political leadership, 
whereas those of the close-ended format considered vio- 
lence as most important [16]. 

From these data, we wanted to evaluate how clinical 
vignette-based surveys influence physician responses. 
Assuming that closed-ended (multiple-choice) questions 
for vignettes produce different responses than open- 
ended, leading to an overestimation of professional per- 
formance, we aimed to determine whether the influence 
of deceptive response items included in the closed-ended 
questionnaires result in a better assessment of profes- 
sional performance. 



Methods 

We conducted a prospective randomized study aimed at 
comparing three response modalities for a vignette-based 
survey: open-ended questionnaire, closed-ended ques- 
tionnaire (with only correct response items) and closed- 
ended questionnaire with deceptive response items mixed 
with correct items. 

Survey 

The survey was composed of two parts. The first part was 
short, identical in each questionnaire, and collected 
demographic characteristics and specialties of physicians. 
The second part was the clinical vignette. 

Vignette 

The vignette reported the history of a fictitious 50-year-old 
woman with active rheumatoid arthritis, a candidate for 
therapy with tumor necrosis factor (TNF) blocking agents. 
Physicians were asked to answer four questions about 
their pre-treatment assessment, considering that TNF- 
blocking treatment was planned: 1 ) what specific data are 
you searching for in this patient's history? 2) What clinical 
data are you personally searching for during the physical 
examination? 3) Which biological, radiographic or other 
tests do you request? 4) What other preventive measures 
do you take? Physicians were given these questions in one 
of three questionnaire formats: open-ended questionnaire 
(questionnaire A) [see Additional file 1], closed-ended 
(multiple-choice) questionnaire with deceptive response 
items mixed with correct items (n = 73) (questionnaire B) 
[see Additional file 2], closed-ended questionnaire with 
only correct items (n = 35) (questionnaire C) [see Addi- 
tional file 3]. Deceptive and correct response items were 
created by following published international and national 
recommendations to help physicians care for patients 
under this treatment [17-21]. Three experts (XM, TP and 
FL) met to formulate correct and deceptive items. They 
based their work on the published international and 
national recommendations to help physicians care for 
patients under this treatment, to first determine the cor- 
rect items, and then propose deceptive items. Each expert 
has elaborated 20 deceptive items, within 4 categories: 
patient's history, physical examination, biological, radio- 
graphic or other tests and other preventive measures. 
From the 53 elaborated items (duplicates were elimi- 
nated), only the more believable were kept, allowing to 
propose 38 deceptive items, which were mixed with the 
35 correct items in questionnaire B. 

Scoring 

Responses to questionnaire A were coded for comparison 
to those of the other two questionnaires. For each item, 
the response was classified as "item correctly selected"; 
"item incorrectly selected"; "item correctly not selected"; 
"item incorrectly not selected." We classified each item 
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according to three sources: an evidence-based literature two checklists of items for pretreatment assessment for 
search of clinical practice concerning TNF-blocking drug TNF-blocker therapy: a long version extracted from the 
management, international and national guidelines, and French clinical tool guide on use of TNF-blocking agents 
a French clinical tool guide on use of TNF-blocking agents [22], with detailed data on research into possible con- 
elaborated by an expert panel of academic and commu- traindications (Table 1), and a short version extracted 
nity physicians [22]. From these sources, we developed from the same clinical tool guide and from the French 

Table I: Long version of the checklist of correct items in pretreatment assessment for TNF-blocker therapy extracted from the 
French clinical tool guide [22]: 

Question I: What specific data are you searching for in this patient's history? 
History of primary tuberculosis infection or of visceral tuberculosis 
History of vaccination with BCG (bacille Calmette-Guerin) 
History of positive response to intradermal tuberculin testing 
History of correct treatment for possible active tuberculosis 
History of infection: cutaneous 



History of infection: pulmonary 
History of infection: urinary 



History of infection: septicemia 



History of infection: septic arthritis 
History of neurologic disease 



History of neoplasia or of hemopathy 



Question 2: What clinical data are you personally searching for during the physical examination? 
Lymphadenopathy 



Neurologic abnormalities 
Question 3: Which biological, radiolographic or other tests do you request? 
Blood count 



Serum protein electrophoresis 



Liver function tests 



Hepatitis C serology 



HIV serology 



Antinuclear antibodies 



Chest x-ray 



Question 4: What other preventive measures do you take? 



Tuberculin skin test 
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agency for health care recommendations with items man- 
datory in France (Table 2) [23]. 

Questionnaire administration 

During the 2004 French Society of Rheumatology meet- 
ing, rheumatologists were asked to participate in a survey 
concerning pretreatment assessment in cases of therapy 
with TNF blockers, which aimed at detecting contraindi- 
cations to treatment. The survey was conducted on behalf 
of the Club Rhumatismes et Inflammation (CRI), the divi- 
sion of the French Society of Rheumatology dedicated to 
musculosquelettal inflammatory diseases. 

Until the targeted sample size was achieved, the survey 
distribution was randomized, with each physician receiv- 
ing only one questionnaire format (A, B or C). Rheuma- 
tologists were blinded to the hypothesis. Particularly, they 
were unaware of the existence of different response 
modalities, of deceptive items in questionnaire B, and that 
all items of questionnaire C were correct. Time to com- 
plete the survey was limited to fifteen minutes. 

Four interviewers were responsible for encouraging partic- 
ipation in the survey, explaining the official nature of the 
survey, checking that all the questionnaires were correctly 
completed in the time allowed, and checking the rand- 
omization was achieved. Participation was voluntary, and 
physicians' responses were kept anonymous. 

Statistical analysis 

A chi-square test was used to compare the proportion of 
items selected or not in terms of the questionnaire format 
of each of the three questionnaires. A p < 0.05 was consid- 
ered statistically significant. Pairwise chi-square tests with 
Bonferonni correction (corrected significant probability 
of 0.017) were used to compare variables with statistically 
significant differences. Statistical analysis involved use of 
SAS Release 8.2 and Splus 6.2. 



Sample size calculation: Three sets of 100 questionnaires 
- one set for each of the three questionnaires - were 
planned for the analysis. In fact, when considering pair- 
wise comparisons for the response item "tuberculin skin 
test, " with a sample size of 100 in two groups, a two-group 
chi-square test with a 0.017 two-sided significance level 
would have 80% power to detect a difference between a 
65% proportion in one group and a 85% proportion in 
the other group. Because we expected 15% incomplete or 
non-analyzable questionnaires, we distributed 350 ques- 
tionnaires. 

Results 
Respondents 

Of 350 questionnaires dispensed (114 questionnaire As, 
118 questionnaire Bs and 118 questionnaire Cs), all were 
completed, and all responses were eligible for further 
analysis. Table 3 displays demographic and specialty char- 
acteristics of physicians responding to the identical format 
part of the survey. Physicians were similar in terms of sex, 
practice duration and practice modalities. Questionnaire 
A respondents were younger than those of the other two 
questionnaires. Only two questions were asked about 
rheumatologists 1 experience with TNF-blocking drugs: 
69.4% had already prescribed anti-TNF therapy and 
43.1% had access to a checklist for screening potential 
contraindications in their department. 

Questionnaire responses 

Although we expected 15% incomplete or non-analyzable 
questionnaires, we did not observe any missing data for 
open-ended or closed-ended questionnaires. 

Significant differences depending on questionnaire for- 
mat were found in reporting pre-treatment assessment. 
Compared with the two closed-ended questionnaires, the 
open-ended questionnaire gave lower reporting of items 
correctly selected and correctly not selected (Table 4). 



Table 2: Short version of the checklist of correct items in pretreatment assessment for TNF-blocker therapy extracted from the 
French agency for health care recommendations (mandatory in France)[23]: 



Question I: What specific data are you searching for in this patient's history? 



History of tuberculosis 



Question 2: What clinical data are you personally searching for during the physical examination? 



None 



Question 3: Which biological, radiographic or other tests do you request? 



Chest x-ray 



Question 4: What other preventive measures do you take? 



Tuberculin skin test 
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Table 3: Demographic characteristics and specialties of physicians completing questionnaires A, B and C. 



Physician characteristics 


Questionnaire A 
(open-ended) 


Questionnaire B 
(closed-ended with deceptive 
items) 


Questionnaire C 
(closed-ended) 


IN 


1 1 A 
1 1 4 


1 1 Q 
1 1 o 


i i q 
I I o 


Sex (% female) 


41.2 


37.3 


37.3 


Age (mean ± SD) 


43.8 ± 9.2 


46.4 ± 8.8 


47.2 ± 9.6 


Years practicing rheumatology (mean ± SD) 


17.8 ±8.6 


18.7 ±8.5 


19.4 ±8.9 


Exclusive private practice (%) 


35.0 


32.8 


31.1 


Practice Both private and public 
practice (%) 


26.5 


32.8 


28.6 


Exclusive public practice (%) 


38.5 


34.4 


40.3 


Previously prescribed TNF-blocking drug (yes, %) 


64.9 


68.6 


73.9 


Access to a checklist before TNF-blocker drug 
prescription (yes, %) 


46.6 


44.5 


51.9 



Questionnaire A: open-ended; Questionnaire B: closed-ended with deceptive items; Questionnaire C: closed-ended 



In terms of global results, none of the questionnaire A 
respondents proposed response items of the long check- 
list, although 5.0% and 5.9% of questionnaire B and C 
respondents, respectively, correctly selected all items 
(Table 4). 

For the A, B and C questionnaires, 50.4%, 84.0% and 
95.0% respondents, respectively, correctly selected all 
mandatory response items of the short checklist. When 
focusing on response items within the short checklist, 
questionnaires B and C did not produce differences in 
responses to item "order chest X-rays" also difference was 
observed with open- and close-ended questionnaires (p < 
0.0001) (Table 4). In contrast, respondents to question- 



naires A, B, and C significantly differed in responses for 
another mandatory item, "obtaining a tuberculin skin 
test": 65.8%, 85.7% and 95.8% respondents, respectively, 
identified this item. 

Rheumatologists completing the closed-ended question- 
naire B, with deceptive response items, more often chose 
these items, such as seeking advice of a systematic lung 
specialist (26.1%) or determining blood sugar level 
(40.3%). None of the questionnaire A respondents spon- 
taneously proposed these items. Questionnaire B 
respondents showed a tendency for a lower percentage of 
correctly selected items than questionnaire C respond- 
ents. The open-ended format allowed for collecting qual- 



Table 4: Percentage of physicians who correctly selected all the items of the short or long checklist for pretreatment assessment for 
TNF-blocker therapy and to specific items, by questionnaire A, B and C and pairwise chi-square comparison 





Questionnaire A 


Questionnaire B 


Questionnaire C 


A and B comparison 


A and C 


B and C 


A, B and C 










p-value 


comparison 


comparison 


comparison 












p-value 


p-value 


p-value 


Long version of checklist 


0.0 


5.0 


5.9 


0.0293 


0.0143 


1.0000 


0.0167 


Short version of 


50.4 


84.0 


95.0 


<0.000l 


<0.000l 


<0.000l 


0.0060 



checklist 



Chose item "chest X- 84.6 96.6 98.3 0.0015 0.0002 0.6835 <0.000l 

rays" 



Chose item "tuberculin 65.8 85.7 95.8 0.0004 < 0.0001 0.0072 O.000I 

skin test" 



Questionnaire A: open-ended; Questionnaire B: closed-ended with deceptive items; Questionnaire C: closed-ended 
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itative data on items that we did not propose in the close- 
ended questionnaires, such as "give information to the 
patient on potential adverse effects" or "give information 
to the patient on monitoring these drugs." 

Discussion 

We compared three clinical vignette-based survey 
response formats: an open-ended questionnaire, a closed- 
ended (multiple-choice) questionnaire with cued correct 
items and a closed-ended questionnaire with deceptive 
items mixed with correct items. As expected, use of a 
closed-ended questionnaire with cued items overesti- 
mated physicians' performance as compared with an 
open-ended questionnaire, given that the latter is consid- 
ered as the gold standard in assessing practice [5,12,24]. 
Also as expected, the open-ended questionnaire supplied 
more information on clinical practice than the close- 
ended questionnaires; physicians were more willing to 
provide information to the patient. Although we included 
response items on examinations or tests, such as cutane- 
ous examination, in the closed-ended questionnaires, 
none of the respondents of the open-ended questionnaire 
suggested such tests. 

Our study focuses on the difficulty in evaluating the qual- 
ity of physician performance for specific domains with 
open-ended questionnaires. Physicians may be more brief 
with open-ended formats and responses may be less accu- 
rate. Of the 114 questionnaire A respondents, 74.4% 
responded with "tuberculosis" to the "other tests" ques- 
tion but gave no specific description of a test or what clin- 
ical examination they would do to evaluate this 
tuberculosis risk. In the closed-ended format, we assumed 
that including deceptive items would influence respond- 
ents' answers and lower the overestimation inherent in 
the closed-format survey. To our knowledge, this is the 
first time that deceptive items have been mixed with cued 
items in a close-ended questionnaire. Questionnaire B 
respondents indeed selected fewer correct items than did 
questionnaire C respondents. However, these results were 
very different from those obtained with the open-ended 
questionnaire (A), which are probably closer to reality. 

The influence of framing questionnaire items remains cru- 
cial for clinical practice evaluation. This bias in response 
acquiescence has been reported from study of two ver- 
sions of a training satisfaction questionnaire randomly 
distributed to medical residents; in one, half the items 
were stated positively and half negatively, and in the 
other, all items were stated positively. Results showed a 
significant effect of positive versus negative framing [25]. 

Conclusion 

In conclusion, even if closed-ended questionnaires may 
provide more accurate data in clinical practice evaluation, 



general open-question format has value in such evalua- 
tion. Strategies for generating quantitative and qualitative 
data from open-ended questionnaires, associated or not 
with closed-ended questionnaires, facilitating survey anal- 
ysis, are very likely interesting to develop to improve phy- 
sician performance evaluation [25]. 
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